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(57) Abstract: An audio system for providing an immer- 
sive audio experience, the system comprising: a recording 
system (21) including a system for recording spatialised au- 
dio information (23); a broadcasting system (30) for broad- 
casting the spatialised audio information to multiple users; a 
series of rendering systems (38) for rendering the broadcast 
spatialised audio information in a manner so as to maintain 
the spatialisation in the presence of movement of a listener's 
head. Preferably, the recording system records video infor- 
mation associated with events occurring contemporaneously 
with the recording of the spatialised audio information and 
the rendering system preferably can include rendering the 
video information to the listener. The system can be utilised 
to broadcast an immersive audio experience associated with 
a sporting event. Multiple different spatialised audio infor- 
mation. The audio tracks can include a commentary of the 
immersive audio experience. 
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VIRTUAL AUDIO ENVIRONMENT 

Field of the invention 

The present invention relates to the field of Audio Signal Processing, and, in particular, 
discloses an immersive audio application having novel operational characteristics. 

5 Background of the invention 

It is well known to provide a system for remote viewing of events such as sporting 
contests or alike. Systems presently available include radio broadcasts and television 
broadcasts of events. Television broadcasts normally consists of a video and audio track with 
the video track taken from a camera and the audio track taken from microphones placed around 
10 the event. Different forms of broadcasting such events are possible. For example, recently, 
Internet streaming has become a popular form of broadcast format. Further, compressed forms 
of broadcast transmission are becoming popular especially with digital compression and 
decompression technologies. For example MPEG video is often a very popular form of video 
compression technology. 

15 It will be desirable if a more immersive form of broadcast of an event was possible. 

Summary of the invention 

It is an object of the present invention to provide an alternative form of immersive audio 
and visual broadcasting system. 

In accordance with a first aspect of the present invention, there is provided an audio 
20 system for providing an immersive audio experience, the system comprising: a recording system 
including a system for recording spatialised audio information; a broadcasting system for 
broadcasting the spatialised audio information to multiple users; and a series of rendering 
systems for rendering the broadcast spatialised audio information in a manner so as to maintain 
the spatialisation in the presence of movement of a listener's head. 

25 The client units further can comprise audio feedback units for providing audio feedback 

to the broadcasting system; and the broadcasting system further including means for forwarding 
audio feedback from first the client units to other ones of the client units for rendering by the 
rendering system. 
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The recording system can be located at a sporting event and the broadcasting system 
forwards the audio feedback to the sporting event for playback at the sporting event. The 
recording system can record video information associated with events occurring 
contemporaneously with the recording of the spatialised audio information and the rendering 
5 system preferably can include rendering the video information to the listener. 

Multiple different spatialised audio tracks are preferably recorded and mixed together to 
form the spatialised audio information. The audio tracks can include a commentary of the 
immersive audio experience. The broadcasting can comprise one of radio transmission, cable 
transmission or Internet Streaming. 

10 The spatialised audio information can comprise B-format audio information or the 

spatialised audio information can comprise a spatialisation of AC-3 encoded audio information. 

In accordance with a further aspect of the present invention, there is provided a method 
of creating an immersive audio experience comprising the steps of: (a) recording an event in a 
spatialised audio format; (b) transmitting the recording to a series of listener stations; and (c) 
15 rendering the spatialised audio to the listeners in a spatialised manner. 

Preferably, the method also comprises the steps of: (d) recording a series of listener 
audio inputs at a plurality of listener stations; (e) transmitting the listener audio inputs to 
predetermined listener stations; (f) rendering the listener audio inputs at the predetermined 
listener stations in conjunction with the spatialised audio. Preferably, the method also comprises 
20 the step of: (g) forwarding predetermined listener audio inputs to the events location for 
playback over an event audio system. 

Brief description of the drawings 

Notwithstanding any other forms which may fall within the scope of the present 
invention, preferred forms of the invention will now be described, by way of example only, with 
25 reference to the accompanying drawings in which: 

Fig. 1 illustrates an example output environment of the preferred embodiment; 

Fig. 2 illustrates a schematic block diagram of the operational portions of an 
embodiment; 

Fig. 3 illustrates a schematic block diagram of an alternative embodiment; and 
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Fig. 4 illustrates schematically the client unit of Fig. 3 in more detail. 

Detailed description of the embodiments 

In the preferred embodiment, there is provided an audio visual immersive system having 
sound spatalization characteristics which provides for an enhanced audio experience. 

5 Turning to Fig. 1, there is illustrated an initial example viewing arrangement 1 of the 

preferred embodiment wherein a listener 2 wears a pair of head tracked headphones 3 which 
include a head tracking unit 4 which tracks the current orientation of the headphones 3 and, in 
turn, the listener's head 5. The listener views a sporting event 7 on a monitor whilst listening 
to an audio track via the headphones 3. The audio track provides spatialization of the listening 

10 environment such that audio sources appear to be spatalized around a listener at pre-determined 
locations. The spatalization is such that the spatalization remains substantially constant within 
the reference frame of the monitor 7 when the listener 2 rotates their head 5 (i.e. the spatial 
location of sound sources to the listener remains constant as the listener rotates their head). 

Turning to Fig. 2, there is a schematic block diagram of one form of system suitable for 
15 implementation of the preferred embodiment. In this system, there is firstly provided, at a 
sporting event or the like, a video capture system 20 for capturing at least one video 
representation of the event. The video capture may be mixed or that taken from a single 
camera. Further, depending on the encoding system utilised, multiple video representations can 
be independently captured. 

20 Also at the event 21 is one or more B-format microphones 23. The B-format 

microphones are able to capture a three dimensional representation of the spatialization of an 
audio environment. The B-format microphone, can, for example, be strategically spaced in an 
ideal viewing area at the event. Additionally, other audio information can also be provided 24 
in a B-format output signal. This allows for the characterisation of audio sources at particular 

25 locations around a listener. For example, a commentary of the occurrence of events at the 
sporting event can be provided with the commentator being located quite close to the shoulder 
of the listener/viewer of the event. Systems for rendering audio signals to particular locations in 
B-format are known. The B-format signals are mixed together 26. The mixing can be a quite 
simple addition of the B-format components. Alternatively, more complex mixing schemes 

30 could be utilised. The audio output in addition to the video output is forward to a compression 
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or formatting system 27 which compresses the streams of information into a compressed 
format. Obviously, other forms of streams can be simultaneously compressed as part of the 
output signal and, for example, multiple B-format signals and other associated audio tracks in 
addition to other video signals and associated visual information such as players statistics could 
5 also be compressed into the output compressed format. 

The output of the compression formatting system 27 is forwarded 29 to a broadcasting 
system 30, The broadcasting system could comprise, for example, an Internet type streaming 
environment or form alternative more standard forms of broadcasting such as those utilised in 
the broadcasting of television such as radio signals, satellite transmission, cable transmission 
10 etc. The broadcast can be from a single broadcasting station to multiple users. 

Each user in receipt of the signal includes a decompression system 32 which 
decompresses the compressed video and audio streams to produce audio B-format output 33 
and video output 34. The video output 34 is forwarded to a video reproduction system 36 
where it is conditioned become video output on the screen display 7 (Fig. 1). The audio output 

15 stream is forwarded to a B-format reproduction system 37 which takes, as part of its input, a 
current user head orientation 38 and produces audio output 39 so as to maintain a substantially 
constant spatalization of audio B-format in the presence of head movements of a listener. The 
B-format reproduction system can be constructed substantially in accordance with the system 
disclosed in the United States Patent No. 6021206 entitled "Method and Apparatus for 

20 processing spatalized audio", the contents of which are specifically incorporated by cross 
reference. The system disclosed in the aforementioned patent specification allows for 
spatalization of B-format audio in the presence of head movements of a listener. 

Hence, the arrangement of the preferred embodiment provides for an immersive audio 
environment such that the user is placed in one of the "best seats in the house" and provided 
25 with a fully immersive audio environment. 

Many further modifications are possible to preferred embodiment. For example, 
multiple users can be accommodated with substantial reuse of components through the 
utilisation of multiple B-format reproduction system units 37 with each user having an 
independent user head orientation. Obviously, only one B-format signal need be broadcast so as 
30 to cater for multiple independent listeners. 
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Further, other encoding arrangements can be provided. For example, where a 5 channel 
Dolby AC-3 surround environment audio signal is provided, a speaker encoding can be 
provided transferring the encoding format from say a Dolby AC-3 format to a B-format. The 
surround encoding can also provide for three dimensional characteristics through the height 

5 positioning of sound sources in an AC-3 environment or the adoption of more complex 
environments. Further, more complex formats of video signals could be provided. For 
example, a spherical ball type arrangement (or its digital equivalent) could be provided so that a 
user utilising a pair of goggles can be provided with an even more immersive experience such 
that the user's head orientation 38 is also utilised in the video reproduction system 39 to show a 

10 "field of view" of a current listener's likely viewing orientation. Hence, as the listener rotates 
the head, the image displayed 7 can also be rotated so as to also be in sync with the audio 
rotation. 

The preferred embodiment obvious has many other uses. For example, it provides for an 
enhanced telepresence experience through the increased immersive characteristics through the 
15 utilisation of B-format information. Further, it could be extended to other arrangements such as 
placing the video capture system and B-format microphone on top of a mobile robot or the like 
so as to provide for a high level of telepresence. This will have advantageous operational 
characteristics in harmful or extreme environments or the like so as to provide for enhanced 
telepresence capabilities. 

20 The system described can be further extended to provide multi-user feedback. For 

example, one of the joys of going to a sporting ground and watching an event, is 'debating' the 
on-field action with fellow spectators and friends. Another example of a feedback situation is 
where people often get together to watch a sporting event at one person's home - for example a 
football match being played overseas. 

25 The present invention can be extended to simulate such arrangements so as to provide a 

communications system that allows people to communicate with others that are watching the 
same event. 

Turning to Fig. 3, there is illustrated schematically an example system 40 suitable for 
implementing a feedback type environment. The system includes a number of client units 41- 
30 43. Each client unit is responsible for rending audio sources received from a server 44 in 
addition to forwarding a microphone input to the server 44. The server 44 in turn takes a 
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number of audio feeds from the sporting event. These can include a commentary feed 45, 
various B format sporting venue audio sources 46. Additionally, the system can provide for a 
form of client unit user feedback through a loud speaker system 47 located at the sporting event 
which provides for a form of microphone output of the home users comments. 

5 The server is responsible for forwarding the audio streams to the client units and can 

include means for streaming audio feedbacks from various client units to other client units in 
accordance with user set ups. 

Turning to Fig. 4, there is illustrated in schematic block form one example client unit 
arrangement 41. The arrangement of the client unit 41 can include a line interface 50 which 

10 interfaces with a communications network 51 to the server. The communication can be of a 
standard internet type streaming environment. The audio visual data received from the server 
including the various audio streams is demultiplexed 52 before being decompressed 53. The 
decompressed streams are then forwarded to the spatialisation unit 54 which combines the 
streams in accordance with requirements and spatialised them so as to provide audio outputs 55 

15 which can be fed to headphones placed around the listener. 

Where provided, a microphone input 56 is fed to a mixing and routing unit 57 which 
digitises the input. Subsequently, it is compressed 58 before being multiplexed with other data 
59 and output via line interface 50 to the server. A series of user controls 60 are also provided 
for controlling the overall arrangement 41 so as to provide for a variable spatialisation 
20 experience. 

As multiple client units are able to interact, the system allows people to communicate 
with others watching the same event so as to experience a combined immersive experience. 

Obviously further modifications can be made, for example, rather than headphone 
arrangements, a user's home theatre system could be utilised for spatialising the playback. 
25 Further, various controls could be provided for selective communication and selective 
experiences whereby users only communicate with other people from a nominated group or 
from the ground itself etc. 

Further, the system could allow various sounds such as cheering or applause from people 
listening to the client units to be relayed to the sporting venue and played over the sporting 
30 venues loud speaker system. 
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It would be appreciated by a person skilled in the art that numerous variations and/or 
modifications may be made to the present invention as shown in the specific embodiments 
without departing from the spirit or scope of the invention as broadly described. The present 
embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive. 

5 It will be understood that the invention disclosed and defined herein extends to all 

alternative combinations of two or more of the individual features mentioned or evident from 
the text or drawings. All of these different combinations constitute various alternative aspects of 
the invention. 
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We Claim: 

1. An audio system for providing an immersive audio experience, said system 
comprising: 

a recording system including a system for recording spatialised audio information; 

a broadcasting system for broadcasting said spatialised audio information to multiple 
users; and 

a series of rendering systems for rendering said broadcast spatialised audio information 
in a manner so as to maintain said spatialisation in the presence of movement of a listener's 
head. 

2. A system as claimed in claim 1 wherein said client units further comprise: 
audio feedback units for providing audio feedback to said broadcasting system; 

and 

said broadcasting system further including means for forwarding audio feedback 
from first said client units to other ones of said client units for rendering by said rendering 
system. 

3. A system as claimed in claim 2 wherein said rendering units render said audio 
feedback in a spatialised manner. 

4. A system as claimed in claim 2 wherein said recording system is located at a 
sporting event and said broadcasting system forwards said audio feedback to said sporting event 
for playback at said sporting event 

5. A system as claimed in claim 1 wherein said recording system records video 
information associated with events occurring contemporaneously with the recording of said 
spatialised audio information and said rendering system includes rendering said video 
information to said listener. 

6. A system as claimed in claim 1 wherein said system is utilised to broadcast an 
immersive audio experience associated with a sporting event. 

7. A system as claimed in claim 1 wherein multiple different spatialised audio 
tracks are recorded and mixed together to form said spatialised audio information. 
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8. A system as claimed in claim 7 wherein said audio tracks include a commentary 
of said immersive audio experience. 

9. A system as claimed in any previous claim wherein said broadcasting comprises 
one of radio transmission, cable transmission or Internet Streaming. 

10. A system as claimed in any previous claim wherein said spatialised audio 
information comprises B -format audio information. 

11. A system as claimed in any previous claim wherein said spatialised audio 
information comprises a spatialisation of AC -3 encoded audio information. 

12. A method of creating an immersive audio experience comprising the steps of: 

(a) recording an event in a spatialised audio format; 

(b) transmitting the recording to a series of listener stations; and 

(c) rendering the spatialised audio to said listeners in a spatialised manner. 

13. A method as claimed in claim 12 further comprising the steps of: 

(d) recording a series of listener audio inputs at a plurality of listener stations; 

(e) transmitting the listener audio inputs to predetermined listener stations; 

(f) rendering the listener audio inputs at said predetermined listener stations in 
conjunction with said spatialised audio. 

14. A method as claimed in claim 13 further comprising the step of: 

(g) forwarding predetermined listener audio inputs to the events location for 
playback over an event audio system. 
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